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ABSTRACT 

Various  image  de-aliasiiig  techniques  and  algorithms  have  been  developed  to  improve  the  resolution  of  sensor- 
aliased  images  captured  with  an  undersainpled  point  spread  function.  In  the  literature  these  types  of  algorithms 
are  sometimes  included  under  the  broad  umbrella  of  superresolution.  However,  in  the  current  paper  we  consider 
them  to  be  a  form  of  image  restoration,  because  we  aim  to  restore  image  resolution  lost  due  to  sensor  aliasing,  but 
only  up  to  the  limit  imposed  by  diffraction.  Specifically,  the  work  presented  here  is  focused  on  image  de-aliasing 
using  inicroscanning.  Much  of  the  previous  work  in  this  area  demonstrates  improvement  by  using  simulated 
imagery,  or  using  imagery  obtained  where  the  subpixel  shifts  are  unknown  and  must  be  estimated.  This  paper 
takes  an  experimental  approach  to  investigate  performance,  in  which  the  subpixel  shifts  are  known  a  priori  at 
some  level.  We  use  a  CMOS  camera,  an  ISO  12233  resolution  target,  and  two  linear  translation  stages  which 
provide  two-axis  camera  control  via  RS-232  interface.  The  translation  stages  provide  a  microstepping  capability 
which  allows  discrete  steps  of  approximately  0.1  ^nn.  The  ISO  12233  target  includes  many  features,  including 
those  for  estimating  the  Line  Spread  Function  and  Modulation  Transfer  Function  (MTF)  via  the  tilted-edge 
response  and  Fourier  transform,  as  well  as  sc^uare  wave  burst  patterns  for  estimating  the  MTF  more  directly 
using  the  Contrast  Transfer  Function  method.  This  target  also  includes  a  series  of  slightly  tilted  square  wave 
burst  patterns  intended  for  measuring  the  aliasing  ratio.  We  use  one  of  these  patterns  to  visually  demonstrate 
successful  de-aliasing. 


1.  INTRODUCTION 

This  paper  presents  results  from  our  initial  laboratory  investigation  of  registration  and  de-aliasing  algorithms 
applied  to  imagery  from  a  visible  sensor.  For  the  work  presented  here  we  used  a  PixeLINK  PL-B741U  camera 
based  on  a  CMOS  sensor.  Section  2  provides  necessary  background  on  imaging  theory,  image  registration  and 
de-aliasing  algorithms,  resolution  testing,  and  performance  metrics.  Methodology  and  approach  are  described  in 
Sec.  3.  Section  4  presents  results  based  on  imagery  of  an  ISO  12233  resolution  target  collected  with  the  CMOS 
camera.  This  includes  results  for  object-space  to  image-space  motion  calibration,  which  is  required  because  visual 
inspection  only  yields  an  estimate  of  object  distance.  A  precise  measurement  is  not  straightforward  when  using 
a  compound  lens.  Results  for  registration  and  de-aliasing  algorithm  performance  are  also  presented.  De-aliasing 
was  accomplished  in  both  the  spatial  and  spatial  frequency  domains.  Concluding  remarks  are  provided  in  Sec.  5, 
including  plans  and  recommendations  for  future  work. 

2.  BACKGROUND 
2.1.  Imaging  Theory  Fundamentals 

This  section  presents  a  summary  of  the  standard  imaging  theory  based  on  linear  systems  and  Fourier  optics. 
First,  recall  that  for  coherent  light  the  imaging  system  is  linear  in  complex  amplitude,  whereas  for  incoherent 
light  the  system  is  linear  in  intensity.  For  the  incoherent  case,  the  impulse  response  is  the  squared  modulus  of 
the  amplitude  impulse  response,  and  the  incoherent  image  intensity  is  given  by 


where  ti  is  the  complex  amplitude  (coherent)  impulse  response,  and  Ug  and  ig  are  the  perfect-system  geometric- 
optics  predictions  for  the  image  field  and  intensity,  respectively.  The  impulse  response,  or  Point  Spread  Func¬ 
tion  (PSF),  for  the  incoherent  case  is  given  by  h  =  \h\^.  Equation  1  is  now  written  as 


i  =  h^ig 


(2) 


The  spatial  frequency  spectrum  of  the  incoherent  image  intensity  is  given  by  [1] 


I  =  nig  ,  (3) 

where  H  is  the  Optical  Transfer  Function  (OTF).  Note  that  quantities  in  Eq.  3  are  the  Fourier  transforms  of  their 
normalized  counterparts  (lower-case)  from  Eq.  2:  i,  h,  and  ig  are  normalized  to  unit  volume  before  transforming 
to  Fourier  space.  In  general,  the  OTF  is  a  complex  function.  The  OTF  modulus  is  termed  the  MTF,  while  the 
OTF  phase  is  termed  the  Phase  Transfer  Function, 

The  system  MTF  can  be  represented  as  the  multiplication  of  component  transfer  functions  [2]: 

AIT =  AIT Ffiij X  AIT Fdctcctor  •  (4) 


For  the  work  presented  here  we  assume  no  relative  motion  (during  sensor  integration)  between  the  object  and 
imaging  system,  and  therefore  we  have  ignored  the  contribution  of  motion-induced  blur  to  the  system  MTF. 
We  have  also  ignored  contributions  to  the  overall  MTF  from  the  electronics  and  signal  processing  subsequent  to 
spatial  sampling  and  prior  to  processing  by  de-aliasing  algorithms. 

The  diffraction-limited  OTF  for  a  circular  aperture  with  no  central  obscuration  is  given  by  [l] 
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where  /  =  (/^  +  /2)  is  the  magnitude  of  the  radial  spatial  frequency  vector  and  fc  is  the  frequency  cutoff  for 
an  incoherent  imaging  system  given  by 


(6) 


where  D  is  aperture  diameter,  A  is  (center)  wavelength,  and  Zi  is  the  distance  from  the  exit  pupil  to  the  image 
plane.  For  an  object  at  infinity,  the  incoherent  frequency  cutoff  is  given  by 


fc 


1 

Jf 


(7) 


where  F  is  the  optical  system  focal  ratio  or  /-number. 

The  transfer  function  describing  the  spatial  frequency  response  of  pixels  with  a  rectangular  active  area  is 
given  by  [3,4] 

ndctcctori^f X •)  fif}  ^  sine  {dxfx^  sinc  (dy/y)  ,  (8) 

where  dx  and  dy  describe  the  physical  dimensions  of  the  photosensitive  area  within  a  pixel  in  the  horizontal  and 
vertical  directions,  respectively.  The  corresponding  detector  MTF  is 


AIT F detector  —  |shic(dx/x)|  [siiic  (dy/y)|  .  (9) 

Also  note  that  for  a  pixelated  sensor  such  as  a  CCD  or  CMOS,  and  the  case  of  equal  pixel  pitch  along  both  axes, 
the  Nyquist  spatial  frequencies  along  the  two  axes  are  equal  and  given  by 

fxs  —  fys  =  2d~  ’ 

where  dec  is  the  pixel  pitch  (i.e.,  ceiiter-to-ceiiter  spacing  along  the  two  axes). 

The  system  MTF  as  predicted  by  Eqs.  5  and  9  is  presented  as  a  surface  plot  in  Fig.  1(a).  Additionally,  a  cross- 
section  of  the  MTF  along  the  /^-axis  is  presented  in  Fig.  1(b),  including  component  MTFs  describing  diffraction 
and  the  detector  subsystem.  The  detector  Nyquist  frequency  given  by  Eq.  10  is  also  plotted  in  Fig.  1(b)  as  a 
point  of  reference  (vertical  blue  line).  The  plots  shown  in  Fig.  1  were  calculated  using  the  following  parameter 
values:  center  wavelength  A  =  600  nm,  focal  ratio  F/4,  areal  fill  factor  =  40%,  and  pixel  pitch  dec  =  b.7  ^rn. 
These  values  correspond  to  experiment  hardware  described  in  Sec.  3. 


1 


Figure  1.  Analytic  model  of  the  system  MTF  plotted  as  a  surface  in  (a)  and  as  a  cross- sex: t ion  along  the  /r- axis  in  (b). 
The  MTF  was  calculated  using  the  following  parameter  values:  center  wavelength  X  =  600  nm,  focal  ratio  F/4.  areal  till 
factor  =  40%,  and  pixel  pitch  dec  =  6.7  fim.  The  component  MTFs  describing  the  contributions  from  diffraction  and  the 
detector  are  also  plotted  in  (b).  along  with  the  sensor  Nyquist  spatial  frequency  /i.,  =  75  cyc/inm. 


2.2.  Image  De-aliasing  Algorithms 

The  goal  of  cle-aliasing  algorithiris  is  to  compensate  for  image  degradation  resulting  from  inadequate  spatial 
saiiipliiig  by  the  Focal  Plane  Array  (FPA).  Fig.  1(b)  provides  one  illustration  of  this  process;  in  this  scenario  the 
optical  cutoff  frequency  (F^q.  7)  is  fc  =  417  cyc/iiiiii,  which  is  5.6  times  the  Nyquist  frequency.  Therefore,  spatial 
freciuencies  greater  than  75  cyc/mni  and  less  than  417  cyc/inm  are  passed  by  the  optical  system,  but  aliased  to 
lower  frecpiencies  when  captured  by  the  FPA.  Our  present  goal  is  to  mitigate  the  amount  of  aliased  energ>^  in 
the  processed  image. 

To  further  illustrate  the  various  sampling  and  signal  processing  regions,  Fig.  2  presents  two  views  of  the 
relationship  between  the  optical  cutoff  and  sensor  Nyquist  frequencies.  Fig.  2(a)  plots  the  OTF  cutoff  (Eq.  7) 
as  a  function  of  focal  ratio,  as  well  as  the  Nyquist  frequency  (horizontal  blue  line).  These  two  curves  essentially 
divide  the  spatial  frequency— focal  ratio  space  into  3  regions  of  interest.  First,  the  region  below  the  horizontal 
blue  line  corresponds  to  spatial  frequencies  which  are  passed  by  the  optics  and  properly  sampled  by  the  FPA. 
Second,  frequencies  above  the  blue  curve  and  below  the  red  curve  are  passed  by  the  optics,  but  aliased  due  to 
inadequate  spatial  sampling  by  the  FPA.  Third,  the  area  above  the  red  curve  describes  spatial  frequencies  not 
passed  by  the  optics  due  to  diffraction.  In  other  words,  any  useful  frequency  content  in  this  region  generated  by 
post-processing  algorithms  is  termed  superresolutioii. 

Next,  Fig.  2(b)  plots  the  ratio  of  the  OTF  cutoff  frequency  (Eq.  7)  to  the  sensor  Nyquist  frequency  (Eq.  10), 
as  a  function  of  focal  ratio.  This  result  highlights  the  fact  that  de-aliasing  for  a  given  wavelength  range  is  only 
useful  below  a  certain  /-number.  For  this  particular  scenario,  2dcc/^F  >  1  for  F  <  22.3.  Also  note  that  as 
pixel  pitch  is  reduced  there  is  a  decreased  potential  for  benefit  from  de-aliasing.  However,  one  must  also  consider 
SNR  and  data  bandwidth,  which  may  drive  a  given  design  toward  larger  and/or  fewer  pixels.  Therefore,  even 
with  ongoing  improvements  in  sensor  technology  resulting  in  smaller  pixels,  there  may  still  be  applications  and 
scenarios  where  de- aliasing  will  provide  a  valuable  signal  processing  tool  for  improving  image  quality.  Finally, 
note  that  for  a  focal  ratio  of  F/4,  the  ratio  plotted  in  Fig.  2(b)  is  equal  to  5.6.  In  terms  of  our  experimental 
setup,  the  aliasing  ratio  will  be  somewhat  less  that  5.6  due  to  factors  such  as  optical  aberrations  and  a  finite 
object  distance  (see  Eq.  6).  Therefore,  we  chose  1/4  pixel  for  an  initial  microscaiining  step  size. 

The  microscaniiing  approaches  described  here  basically  use  a  sequence  of  frames  recorded  with  known,  or 
estimated,  subpixel  image  shifts.  In  general  the  shifts  are  unknown,  and  a  registration  algorithm  becomes  a  key 
component  to  successful  de-aliasing.  In  terms  of  algorithm  design,  de-aliasing  can  be  approached  from  both  the 
image  and  spatial  frequency  domains.  In  the  image  domain  the  subpixel  shifts  are  essentially  used  to  directly 
generate  a  High  Resolution  (HR)  grid.  Although  in  some  i  in  age-domain  approaches,  the  HR  image  intensities 
are  obtained  by  solving  a  constrained  miniiiiization  problem  [4].  On  the  other  hand,  the  set  of  Low  Resolution 
(LR)  sub  pixel-shifted  images  can  be  transformed  to  spatial  frequency  space  using  the  FFT,  and  then  used  to 
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Figure  2.  Two  illustrations  of  the  sampling  and  imago  processing  regions  of  interest.  Fig.  (a)  shows  the  OTF  cutoff 
frequency  (Eq.  7)  as  a  function  of  focal  ratio,  as  well  as  the  Nyquist  frequency  (horizontal  blue  line).  These  values  are 
based  on  a  wavelength  of  A  =  600  nm  and  a  pixel  pitch  of  dec  =  6.7  //m.  Fig.  (b)  shows  the  ratio  of  OTF  cutoff  frequency 
to  the  sensor  Nyquist  frequency,  as  a  function  of  focal  ratio. 


form  a  system  of  linear  equations  (for  each  LR  pixel).  This  set  of  linear  systems  can  then  be  solved  using 
linear  algebra  techniques  to  unscramble  the  aliased  frequency  content  [5-7].  The  inverse  Fourier  transform  of 
this  “unscrambled”  spectrum  yields  the  desired  de-aliased  image.  This  technique  is  described  by  the  following 
equation: 

y  =  $x  ,  (11) 

where  y  is  a  column  vector  of  image  DFT  coefficients,  $  is  a  matrix  of  complex  exponential  factors  relating  the 
DFT  coefficients  of  the  shifted  images  to  coefficients  of  the  aliased  (i.e.,  overlapping)  copies  of  the  Continuous 
Fourier  Transform  (CFT),  and  x  is  a  column  vector  of  unknown  CFT  coefficients.  Values  for  y  are  taken  from 
the  set  of  2D  FFTs  of  the  image  sequence,  and  the  matrix  ^  is  generated  based  on  knowledge  of  the  subpixel 
shifts  and  an  assumed  aliasing  ratio.  Note  that  Eq.  11  is  solved  for  each  point  in  the  LR  grid,  and  each  solution 
yields  a  set  of  reconstructed  CFT  coefficients  which  are  then  placed  in  the  HR  FFT  grid. 

2.3.  Image  Registration  Algorithms 

A  wide  variety  of  image  registration  algorithms  have  been  investigated  and  developed  over  the  past  several 
decades.  Several  algorithms  were  utilized  during  the  course  of  this  effort,  to  include: 

•  The  basic  phase  correlation  technique  [8] 

•  The  modified  phase  correlation  technique  for  subpixel  registration  described  by  Foroosh,  et  al.  [9] 

•  A  subpixel  registration  technique  described  by  Guizar-Sicairos,  et  al.  [10]  (MATLAB®code  available  on-line) 

The  first  algorithm,  the  basic  phase  correlation  technique,  only  provides  a  shift  estimate  to  the  nearest  integer 
number  of  pixels.  Therefore,  this  technique  is  not  useful  for  de- aliasing,  but  does  provide  a  foundation  for  the 
development  of  other  algorithms.  The  following  two  algorithms  do  provide  a  subpixel  shift  estimate.  For  this 
work,  we  assume  only  a  global  image  translation.  The  original  image  intensity  is  described  by  /(x,^),  and  the 
shifted  image  is  g{x^y)  =  f{x  —  xo,  y  —  yo)-  The  phase  correlation  technique  and  its  variations  are  based  on  the 
normalized  cross  power  spectrum: 


j7(|mf)1=exp{27rj(/.Xo  +  /,j/o)} 


(12) 


After  computing  the  normalized  cross  spectrum  given  by  the  left  side  of  Eq.  12,  the  resulting  (ideally)  planar 
phase  modulation  is  transformed  back  to  the  image  domain  via  the  inverse  FFT.  The  location  of  the  resulting 


peak  provides  an  estimate  of  the  phase  tilt,  which  corresponds  to  an  estimate  of  the  image  shift.  The  approach  of 
Foroosh  [9]  is  based  on  a  multi-rate  signal  processing  formulation  of  the  phase  correlation  teclnhqne.  Essentially, 
this  method  involves  fitting  a  Diiichlet  function  to  the  peak  pixel  and  its  neighbors.  The  simplified  approach 
to  this  fitting  as  presented  by  Foroosh  is  very  straightforward  to  implement;  only  a  few  additional  lines  of  code 
are  required  once  the  basic  phase  correlation  technique  has  been  implemented.  Finally,  the  approach  of  Guizar- 
Sicairos  [10]  is  a  very  efficient  implementation  of  Fourier  interpolation,  which  is  used  to  refine  the  estimate 
produced  by  the  basic  phase  correlation  technique. 

2.4.  Optical  System  Resolution  Targets  and  Testing 

A  variety  of  techniques  and  standardized  targets  have  been  developed  and  utilized  for  testing  the  resolution  of 
optical  systems.  Some  of  these,  such  as  the  USAF  1951  Target  and  the  NBS  1952  Target,  are  based  exclusively  on 
a  set  a  3-  or  4-bar  patterns  where  each  pattern  is  of  a  certain  spatial  frequency.  This  type  of  target  can  be  used  to 
infer  the  limiting  spatial  frequency  by  visual  inspection,  or  to  estimate  the  value  of  the  MTF  at  several  different 
spatial  frequencies  by  way  of  the  Contrast  Transfer  Function  (CTF).  Another  approach  to  resolution  testing  is 
through  the  use  of  random  test  patterns.  This  idea  is  analogous  to  the  use  of  a  white  noise  pattern  in  acoustics  to 
determine  system  frequency  response,  as  well  as  the  “FFT  method”  of  generating  atmospheric  phase-screens  [ll] 
for  wave-optics  simulations.  For  this  work  we  used  the  ISO  12233  target  developed  for  high  definition  television 
(HDTV)  applications  [12|.  The  ISO  12233  target  includes  several  slanted  knife-edge  targets  for  the  purpose  of 
computing  the  MTF  by  Fourier  transform.  They  are  offset  from  the  vertical  or  horizontal  by  5  degrees  so  that 
the  edges  will  be  sampled  at  many  different  phases,  which  allows  estimation  of  the  spatial  frequency  response 
beyond  the  Nyquist  frequency  of  the  sampling  (etjuivalent  to  the  moving  knife  edge  approach). 

2.5.  Performance  Metrics 

Typical  metrics  used  to  characterize  the  resolution  of  an  imaging  system  include: 

•  Optical  Transfer  Function  [SPATIAL  FREQUENCY  DOMAIN] 

—  Modulation  Transfer  Function 
—  Phase  Transfer  Function 

•  Limiting  Resolution  (at  a  specified  contrast)  [SPATIAL  FREQUENCY  DOMAIN] 

•  Resolution  [IMAGE  DOMAIN] 

—  Rayleigh 
—  Sparrow 

For  some  applications  and  object  scenes  of  interest,  the  limiting  resolution  may  not  be  the  best  metric  for 
describing  system  performance.  The  shape  of  the  MTF  curve  below  or  above  the  limiting  resolution  may  be 
of  greater  importance.  Ultimately,  the  spatial  frequency  content  of  the  scene  determines  the  importance  of  a 
system’s  ability  to  resolve  a  particular  spatial  frequency. 

3.  METHODOLOGY  AND  APPROACH 

3.1.  Experiment  Setup 

A  PixcLINK  CMOS  camera  was  u.scd  to  investigate  algorithm  performance.  Key  specifications  for  this  camera 
are  provided  in  Tab.  1.  An  Edmund  Optics  Focusable  25  mm  Double  Gauss  Lens  (#NT55-326)  was  used  with 
the  PixeLlNK  camera.  This  lens  has  a  miiiinium  working  distance  of  240  mm,  and  a  niaiinally  adjustable 
aperture  allowing  focal  ratios  of  F/4  to  oo.  The  design  MTF  for  the  lens  is  approximately  equal  to  0. 1  at  a 
spatial  frequency  of  300  cyc/nnn.  This  is  approximately  4  times  the  sensor  Nyquist  limit,  which  guides  our 
choice  of  1/4  pixel  for  the  initial  in icro.se aiming  step  size.  Significant  resolution  improvement  is  not.  expected  to 
result  from  inicroscan  increments  considerably  smaller  than  1/4  pixel,  as  diffraction  becomes  the  limiting  factor. 
An  Edmund  Optics  Enhanced  Digital  Camera  Resolution  Chart  (#NT58-940)  was  used  as  a  test  object.  This 
chart  complies  with  the  ISO  12233  standard.  To  implement  a  niicroscanning  capability  for  these  experiments, 
two  linear  motion  stages  were  combined  to  allow  computerized  x-y  motion  control  of  the  PixeLlNK  camera  via 
RS-232  interface.  More  specifically,  we  used  two  Zaber  Linear  stages  (KT-LS28-I)  and  a  90  degree  angle  bracket 
(AB90-I).  According  to  the  Zaber  documentation,  the  distance  per  microstep  is  0.09921875  f.iin.  A  tyjiical  setup 
with  the  PixeLlNK  camera  attached  to  the  Zaber  stages  is  shown  in  Fig.  3. 


Table  1.  Key  specifications  for  the  PixeLINK  PL-B741U  CMOS  camera. 


PixeLINK  PL-B741U 

Spectral  Response 

visible  (400-700  nm) 

Resolution 

1280x1024  pixels 

Frame  Rate 

27  Hz  (full  frame) 

Sensor  Type 

CMOS 

Bit  Depth 

8  or  10 

Pixel  Pitch 

6.7  /nn 

Fill  Factor 

40% 

Interface 

USB  2.0 

Figure  3.  A  typical  experiment  setup  with  the  PixeLINK  camera  attached  to  two  microstepping  linear  translation  stages. 

3.2.  Data  Collection  Plan 

Once  the  camera,  linear  motion  stages,  and  ISO  12233  resolution  target  were  installed,  several  data  sets  were 
taken  sequentially  to  accomplish  both  actuator-to-iniage  motion  calibration,  as  well  as  registration  and  de-aliasing 
algorithm  testing.  Essentially,  three  distinct  data  sets  were  collected. 

First,  a  series  of  images  was  collected  using  a  set  of  linearly  spaced  positions  spanning  nearly  the  full  range 
of  actuator  motion.  This  was  accomplished  first  for  shifts  along  the  :c-axis,  then  for  shifts  along  the  y-axis. 
Furthermore,  this  data  was  collected  using  both  the  absolute  and  relative  command  modes  allowed  by  the 
control  software  interface  to  verify  correct  operation  of  the  actuators.  This  data  is  used  to  calibrate  the  image 
de-inagnification  factor  due  to  the  fact  that  the  object  distance  is  only  approximately  known  based  on  visual 
inspection.  The  object  distance  measurement  is  not  straightforward  when  using  a  compound  lens. 

Second,  a  series  of  images  was  collected  using  a  set  of  (roughly)  log-spaced  positions  spanning  nearly  the  full 
range  of  actuator  motion.  Again,  separate  data  sets  were  collected  for  x-  and  y-axis  motion.  This  data  was  only 
collected  in  the  absolute  actuator  command  mode.  Given  that  there  are  greater  than  280,000  available  positions 
along  each  axis,  log-spaced  image  positions  were  required  to  provide  a  useable  amount  of  data  while  still  utilizing 
the  full  range  of  motion.  This  data  set  is  used  to  investigate  the  performance  limits  of  the  registration  algorithms. 

Finally,  several  sets  of  microscan  data  were  collected  for  the  purpose  of  demonstrating  de-aliasing  algorithm 
performance.  The  calibration  factor  resulting  from  analysis  of  the  first  data  set  was  used  to  determine  the  number 
of  microsteps  per  image  shift  prior  to  collecting  the  microscan  data.  First,  16  images  were  collected  using  1/4 
pixel  shifts  in  x  and  y.  Second,  36  images  were  collected  using  1/6  pixel  shifts.  These  two  data  sets  then  allow 
de-aliasing  at  1/2,  1/4,  1/3,  and  1/6  subpixel  shifts. 


3.3.  Calibration  Procedure 


A  result  of  our  experiment  setup  is  that  actuator  motion  occurs  in  object  space.  The  observed  image  shift  Axj 
(in  pixels)  is  a  linear  function  of  the  physical  actuator  shift  Axo  (in  microsteps): 


Axi  = 


3 


ft 

Zo  -  II 


Ax„ 


(13) 


where  (5  is  the  physical  distance  per  iiiicrostep  (meters),  dec  is  pixel  pitch  (meters),  is  lens  focal  length  (meters), 
and  Zo  is  object  distance  (meters).  Equation  13  is  utilized  twice  during  the  calibration  procedure.  First,  the 
object  distance  2:0  is  simply  measured  by  visual  inspection.  An  initial  calibration  factor  is  then  calculated  using 
the  slope  term  of  Eq.  13.  This  initial  estimate  is  then  used  to  adjust  the  iiuiiiber  of  niicrosteps  per  image  shift 
as  desired  before  collecting  the  set  of  linearly  spaced  calibration  data  as  described  in  Sec.  3.2.  Once  collected, 
the  calibration  data  is  analyzed  using  one  or  more  registration  techniques,  such  as  those  described  in  Sec.  2.3. 
The  shift  between  each  image  and  the  fiist  image  in  the  sequence  is  estimated.  This  yields  a  set  of  cumulative 
shift  estimates  (in  image  space)  which  are  plotted  as  a  function  of  the  corresponding  actuator  displacement  (in 
object  space).  The  slope  of  a  best-fit  line,  rh,  to  this  data  yields  an  estimate  of  the  desired  calibration  factor. 
Conversely,  this  slope  estimate  can  also  be  used  to  solve  for  an  estimate  of  the  object  distance: 


w;  =  II 


(14) 


based  on  known  parameters  values  from  specifications  for  the  camera,  lens,  and  linear  motion  stages. 


4.  RESULTS 

This  section  presents  results  for  object-space  to  image-space  motion  calibration,  as  well  as  for  performance  of 
the  registration  and  de-aliasing  algorithms.  All  results  presented  herein  are  for  an  object  distance  of  34.5  in. 
(876.3  mm),  as  measured  by  visual  inspection.  Furthermore,  all  images  were  recorded  as  8-bit  moriochroiiie 
bitmaps  using  the  PixeLINK  OEM  software.  The  lens  aperture  was  wide  open  for  a  focal  ratio  of  F/4. 

4.1.  Actuator- to-Image  Motion  Calibration 

Once  the  linear  motion  stages,  camera,  and  resolution  target  were  installed  on  the  optical  table,  the  object 
distance  was  visually  estimated  to  be  z„  ~  876.3  mm.  Using  the  slope  term  of  Eq.  13,  the  corresponding 
calibration  factor  was  found  to  be  2299.4  microsteps/pixel.  This  value  was  then  used  to  choose  5174  microsteps 
per  image  shift,  which  equates  to  approximately  2.25  pixels  per  shift.  This  allowed  collection  of  a  maximum  54 
images  along  each  axis,  as  limited  by  the  28  iiiiii  range  of  the  Zaber  linear  stages.  The  sequence  of  54  images  was 
then  analyzed  (using  the  three  image  registration  algorithms  described  in  Sec.  2.3)  in  order  to  estimate  the  shift 
betw'een  each  image  and  the  first  image  in  the  sequence.  For  the  case  of  relative  mode  actuator  displacement 
along  the  column  direction  only,  the  resulting  set  of  estimates  of  the  cumulative  row-  and  column-directioii  shifts 
are  plotted  in  Figs.  4(a)  and  4(b),  respectively,  as  a  function  of  actuator  displacement. 

Application  of  linear  regression  to  the  data  seen  in  Fig.  4(b)  yielded  slope  estimates  of  422.7  x  10“^,  422.4  x 
10“^,  and  422.5  x  10“^  pixels  per  iiiicrostep  for  the  basic  phase  correlation  technique,  the  techiiiciue  of  Foroosli, 
and  the  technique  of  Guizar-Sicairos,  respectively.  For  these  three  data  sets,  the  residual  fit  error  w’as  least  for  the 
Guizar-Sicairos  registration  algorithm.  Therefore,  this  was  taken  as  the  best  calibration  factor  estimate.  Using 
this  slope  estimate  together  with  Eq.  14,  the  actual  lens-to-object  distance  was  also  estimated  to  be  902  mm. 
The  difference  between  this  value  and  the  original  visual  estimate  is  2.5  cm,  which  is  reasonable.  Also  note  the 
results  of  Fig.  4(a),  which  show  a  slight  upward  trend  in  row  (y-axis)  position.  This  may  be  due  to  an  imperfectly 
balanced  load  and/or  a  slight  iiiiperfection  in  one  or  both  of  the  right-angle  bracket  attachments. 

This  same  process  was  applied  to  calibration  data  for  the  three  remaining  cases:  (1)  absolute  mode  ac¬ 
tuator  displacement  along  colunin-direction,  (2)  relative  mode  actuator  displacement  along  row- direct  ion,  and 
(3)  absolute  mode  actuator  displacement  along  row-direction.  There  was  very  little  variation  in  the  resulting 
calibration  factor  estimates,  and  for  the  remaining  work  presented  in  the  paper  the  calibration  was  take  as 
2368  inicrosteps/ pixel.  This  value  was  used  to  set  the  image  shift  values  prior  to  recording  the  registration  and 
de-aliasing  data  sets. 

Alternatively,  known  dimensions  of  the  ISO  12233  target  can  be  used  to  estimate  the  calibration  factor.  Using 
one  of  the  existing  calibration  images,  the  vertical  size  of  the  active  resolution  target  area  was  estimated  to  be 
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Figure  4.  Estimates  of  global  image  shift  (translation)  for  Relative  Mode  Actuator  Displacement  along  column-direction 
(x-axis)  only.  Results  for  row-  and  column-direction  shift  estimates  are  presented  in  (a)  and  (b),  respectively. 


roughly  855.5  pixels.  The  actual  object  height  is  200  mm,  which  together  with  the  known  physical  distance 
per  microstep  gives  2356  microsteps  per  pixel.  This  is  slightly  smaller  than  the  result  of  the  linear  regression 
approach.  However,  a  more  robust  and  systematic  effort  would  require  an  automated  process  for  estimating  the 
image  edge  location  in  multiple  images,  then  calculating  an  average.  This  effort  was  put  off  for  the  future  since 
the  linear  regression  method  is  robust  and  straightforward  to  implement. 

4.2.  Registration  Algorithm  Performance 

All  three  registration  algorithms  described  in  Sec.  2.3  were  used  to  estimate  the  shift  between  each  image  and 
the  first  image  in  the  sequence  captured  using  log-spaced  actuator  positions.  The  magnitude  of  the  resulting 
cumulative  shift  estimates  are  plotted  versus  the  true  (calibrated)  shifts  in  Figs.  5(a)  and  5(b)  for  x-  and  t/-axis 
shifts,  respectivoly.  Those  results  are  plotted  in  log-log  space  to  highlight  the  effective  limit  of  the  two  siibpixel 
registration  algorithms  considered.  The  technique  of  Foroosh  [9]  is  unable  to  estimate  shifts  less  than  about 
0.2  pixels.  The  technique  of  Guizar-Sicairos  is  able  to  estimate  much  finer  shifts,  but  is  more  computationally 
expensive  and  requires  more  memory.  The  performance  is  controllable,  to  some  extent,  by  choice  of  input 
parameter  a.  However,  running  MATLAB®on  a  WinXP  machine  with  4  GB  of  memory  resulted  in  an  out  of 
memory  error  when  a  was  increased  to  5000.  The  parameter  a  basically  determines  the  extent  of  zero-padding 
used  to  perform  the  Fourier  interpolation.  Another  aspect  of  registration  algorithm  failure  not  clearly  identified 
by  Figs.  5(a)  and  5(b)  is  the  inability  to  identify  the  correct  sign  of  the  shift.  Based  on  our  investigation  to  date. 
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Figure  5.  Estimates  of  cumulative  global  image  shift  (translation)  for  relative  mode  actuator  displacement.  Shift  estimates 
along  the  column  (x-axis)  and  row  (y-axis)  directions  are  presented  in  (a)  and  (b),  respectively. 


(a)  LR  Image  (b)  HR  Image:  Spatial  domain  de-  (c)  HR  Image:  Fourier  domain  de¬ 
aliasing  aliasing 


Figure  6,  Example  of  image  de-aliasing  algorithm  performance  using  1/4  pixel  inicroscan  data.  The  original  LR  image 
is  shown  in  (a),  while  (b)  and  (c)  show  the  result  of  spatial  and  Fourier  domain  de-aliasing,  respectively. 


this  sign  flip  is  mainly  a  problem  for  the  Foroosh  algorithm,  and  becomes  more  prevalent  as  the  shift  becomes 
smaller.  This  could  be  a  serious  issue  for  a  generalized  de-aliasing  algorithm  relying  solely  on  shift  estimates  as 
provided  by  a  registration  algorithm.  In  terms  of  the  data  collected  for  this  work,  this  type  of  error  could  be 
related  to  the  level  of  aliasing  iii  the  image  data.  One  area  worth  more  consideration  is  to  extend  the  method 
of  Foroosli  by  using  a  more  robust  procedure  for  fitting  the  Dirichlet  function  to  the  normalized  cross  spectrum 
image. 

4.3.  De-Aliasing  Algorithm  Performance 

The  calibrated  1/4  pixel  shift  inicroscan  data  set  was  processed  using  both  the  spatial  and  Fourier  domain 
techniciues.  This  resulted  in  a  restored  image  of  a  large  portion  of  the  ISO  12233  target.  To  demonstrate  the 
performance  of  these  algorithms.  Fig.  6  shows  several  versions  of  a  closeup  of  one  of  the  ISO  12233  slanted  scjiiare 
wave  burst  (i.e.,  single  spatial  frequency)  patterns.  The  image  in  Fig.  6(a)  is  from  one  of  16  LR  images  in  the  1/4 
pixel  inicroscan  secpieiice.  The  image  in  Fig.  6(b)  is  from  the  HR  image  produced  using  spatial  domain  processing. 
Due  to  the  use  of  calibrated  motion,  the  spatial  domain  technique  was  implemented  as  simple  interlacing.  The 
image  in  Fig.  6(c)  is  from  the  HR  image  produced  using  Fourier  doinain  processing.  In  both  HR  images,  the 
aliased  spatial  frequency  apparent  in  the  LR  image  has  been  removed.  In  this  particular  example,  the  spatial 
and  Fourier  domain  techniques  yield  a  result  that  is  virtually  identical  based  on  visual  inspection. 

5.  CONCLUSIONS 

This  paper  describes  a  simple  experimental  setup  to  test  the  performance  of  image  registration  and  de-aliasing 
algorithms.  Once  the  calibration  procedure  is  accomplished,  the  number  of  niicrostcps  per  pixel  is  known,  and 
utilized  to  collect  two  additional  data  sets.  The  first  is  a  set  of  images  at  log-spaced  actuator  positions  to 
evaluate  registration  algorithni  performance.  The  second  is  a  set  of  inicroscan  data  covering  a  regular  grid  using 
a  calibrated  subpixel  shift. 

Using  the  log-spaced  data  we  evaluated  the  basic  phase  correlation  technique  for  global  translation  registra¬ 
tion,  as  well  as  two  variations  thereof  allowing  subpixel  shift  estimation.  The  technique  of  Foroosh  was  simple 
to  iniplcnient  with  little  computational  burden  beyond  that  of  the  basic  technique.  However,  for  the  data  wc 
collected  this  technique  was  unable  to  properly  estimate  shifts  smaller  than  about  0.2  pixels.  The  technique  of 
Guizar-Sicairos  was  evaluated  using  the  authors’  code  (a\^ilable  on-line),  and  was  able  to  estimate  much  smaller 
shifts  to  at  least  1/ 100th  of  a  pixel.  This  technique  is  more  computationally  intensive  and  requires  more  memory, 
but  is  somewhat  adjustable  by  an  input  parameter  controlling  the  amount  of  Fourier  interpolation. 

The  inicroscan  data  was  then  processed  using  both  the  spatial  and  Fourier  domain  techniques.  A  visual 
example  of  tlie  result  of  each  technique  was  presented  by  way  of  an  image  of  a  slanted  square  wave  burst  pattern 
on  the  ISO  12233  resolution  target.  This  example  demonstrates  that  both  techniques  have  successfully  mitigated 
the  aliasing  evident  in  the  LR  image. 

Thoughts  and  directions  for  future  work  include  the  following: 


1.  Explore  the  impact  of  the  pixel  fill  factor 


2.  Extend  the  laboratory  setup  to  include  rotation  and  scale  (i.e.,  target  range) 

3.  Explore  the  impact  of  bit-depth  by  using  the  raw  10-bit  PixeLINK  image  format  (*.pds) 

4.  Explore  the  impact  of  scene  complexity,  or  level  of  “information”  content,  on  algorithm  performance 

5.  Investigate  performance  of  the  registration  and  de-aliasing  algorithms  in  the  Long- Wavelength  Infrared  (LWIR) 
using  a  niicrobolometer  camera 

6.  Explore  a  more  robust  fitting  procedure  to  extend  the  registration  algorithm  of  Foroosh 

7.  Explore  other  registration  techniques,  such  as  those  based  on  optimization  to  minimize  a  given  error  metric 

8.  Explore  de-aliasing  algorithm  performance  as  a  function  of  increasing  noise  and  registration  errors.  Does 
this  result  depend  on  whether  the  registration  errors  are  random  or  systematic? 

9.  How  do  registration  errors  compare  with  errors  in  the  de-aliasing  process  itself  in  detenniniiig  the  fidelity 
of  the  de-aliased  image? 

Item  number  4  above  is  related  to  potential  application  of  de-aliasing  to  space-based  imaging.  One  can  imagine 
two  scenarios:  resolved  and  non-resolved.  For  the  resolved  case,  one  can  reasonably  expect  that  registration 
algorithm  technology  combined  with  relative  motion  between  the  object  and  camera  will  provide  the  necessary 
subpixel  shifts,  and  measurements  thereof,  to  implement  de-aliasing.  However,  for  the  case  of  non-resolved 
sensing  the  registration  algorithms  may  fail  due  to  lack  of  scene  content.  There  may  be  an  algorithmic  approach 
to  address  this  issue.  If  not,  additional  hardware  would  be  required  that  would  allow  the  FPA  to  be  actively 
scanned  in  subpixel  increments  in  a  deterministic  fashion.  This  is  related  to  vibration  reduction  technology  used 
in  some  consumer  digital  cameras.  Therefore,  a  straightforward  technology  path  to  this  solution  may  already 
exist.  Furthermore,  the  goal  of  de- aliasing  in  the  resolved  case  is  to  improve  the  resolution  and  accuracy  of  an 
image.  However,  in  the  non-resolved  case  the  probabilities  of  detection  and  false  alarm  are  more  appropriate 
metrics.  This  difference  in  performance  metric  deserves  further  study. 
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